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tions of  ridge  regression  employ  stochastic  techniques  to  select 
the  ridge  parameter.  In  this  paper  we  examine  three  nonstochastic 
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1 . INTRODUCTION 

Hoerl  and  Kennard  (1970a,  b)  introduced  ridge  regression 
estimation  as  an  alternative  to  least  squares  estimation  of  the 
parameters  of  a multiple  linear  regression  model  when  the  predictor 
variables  are  multicollinear . Write  the  regression  model  as 


y = 6 l + xe  + e 


where  Y is  an  n * 1 vector  of  response  variables,  1 is  an  n n 1 

vector  of  ones,  X = [3^,  Xj,  ...»  3M  is  an  n x p full  column  rank 

matrix  of  nonstochastic  predictor  variables  that  are  standardized 

so  that  X'l  « 0 and  X'X.  =1  (j  » 1,  2,  . . . , p) , B and  B are  un- 
-3-3  o - 

known  constants,  and  £ is  an  n * 1 vector  of  random  error  terms 

2 

with  ei  ^ NID (0 , a ) . Hie  least  squares  and  (simple)  ridge  esti- 
mators of  6.  are,  respectively, 

B - (X'X)-1  X'Y  and  £L  - (X'X  + kI)_1X'Y  (1.2) 


where  k > 0. 


Theoretical  properties,  including  optimality  considerations, 
of  ridge  estimators  have  been  derived  under  the  assumption  that  the 
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selection  of  the  ridge  parameter,  k,  is  nonstochastic.  Specifical- 
ly, if  the  (total)  mean  squared  error  of  a regression  estimator  j3 
is  defined  to  be 

mse ($)  « E[(ji. -£)'(£-  £)]  , (1.3) 

Hoerl  and  Kennard  (1970a)  proved  that  a nonstochastic  choice  of  k 
that  is  suitably  small  would  insure  that 

mse(B^)  < mse  (£)  . (1.4) 

If  the  latent  roots  of  X'X  are  denoted  an<*  1:116 

corresponding  latent  vectors  are  V^,  V2,  ...,  V^,  a sufficient 
condition  for  (1.4)  to  hold  is  that 

0 < k < o2/t*  , (1.5) 

where  * maxH^;  j ■ 1,  2,  ...,  p}  and  ♦j  - 

Two  problems  arise  with  the  practical  implementation  of  ridge 
regression  using  (1.5).  First,  the  above  properties  have  only  been 
derived  for  nonstochastic  selection  rules  for  k.  Second,  the  bound 
in  (1.5)  is  a function  of  the  unknown  model  parameters.  Most  of 
the  applications  of  ridge  regression  methodology,  however,  employ 
stochastic  estimators  of  k (e.g.  Hoerl  and  Kennard  1970b;  McDonald 
and  Schwing  1973;  and  Hocking,  Speed,  and  Lynn  1976).  To  date, 
little  attention  has  been  given  to  nonstochastic  selection  rules. 


i 
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2.  THREE  NONSTOCHASTIC  SELECTION  RULES 

One  of  the  motivations  for  adopting  ridge  regression  over 
least  squares  is  that  the  resulting  estimates  behave  more  like  they 
are  generated  by  an  orthogonal  X matrix  than  a multicollinear  one. 
The  rules  proposed  in  this  section  were  investigated  because  they 
enable  (X'X  + kl)  to  exhibit  properties  more  closely  associated 
with  the  identity  matrix  than  does  X'X. 

If  X is  am  orthogonal  matrix  (standardized  as  in  Section  1) , 
the  following  three  properties  are  true  of  X'X: 

(i)  | X'X | - 1 , 

(ii)  (X'X)  1 * I and,  therefore,  the  variance  infla- 
tion factors  (diagonal  elements  of  (X'X) 

Marquardt  (1970))  are  all  equal  to  1 ; and 

(iii)  the  latent  roots  of  X'X  are  all  equal  to  1 (i.e., 
Jtj  * 1 for  all  j)  . 

These  characteristics  all  follow  from  the  realization  that  X'X  - I 
if  the  columns  of  X are  mutually  orthogonal.  On  the  other  hand, 
if  X contains  highly  multicollinear  colvrans  (e.g.  Mason,  Gunst, 
and  Webster  1975) , 

(i)  |x'x|  = 0 } 

(ii)  the  variance  inflation  factors  are  much  larger 
than  1;  and 

(iii)  one  or  more  of  the  latent  roots  of  X'X  are  close 


to  0 


In  order  to  compensate  for  the  deleterious  effects  of  multi- 
collinearities  on  these  properties  of  X'X,  we  propose  to  examine 
the  following  three  rules  for  selecting  k.  The  resulting  ridge 
regression  estimators,  all  of  which  are  found  using  8 in  (1.2), 

K 

are  identified  as  RR(2),  RR(3),  and  RR(4) : 

RR(2) : Choose  k so  that  |x'X  + kl | =1. 

RR(3):  Choose  k so  that  the  largest  variance  inflation  factor 
equals  4. 

RR(4):  Choose  k so  that  the  "multicollinearity  allowance" 
(Vinod  1976) 

P 

»*P-  I +k)  1 (2.1) 

j-1  1 Z 

equals  the  number  of  multicollinearities  existing 
among  the  predictor  variables. 

The  multicollinearity  allowance,  m in  (2.1),  can  be  inter- 
preted (Vinod  1976)  as  "the  assigned  deficiency  in  the  rank  of 
X'X."  In  our  investigations  if  X is  an  orthogonal  matrix,  or 
nearly  so,  a rank  deficiency  of  zero  is  assigned  to  X'X.  Solving 

(2.1)  for  k when  m ■ 0 yields  k * 0;  i.e.,  the  ridge  estimator  in 

(1.2)  reduces  to  the  least  squares  estimator.  On  the  other  hand, 
if  r multicollinearities  are  identified  in  X,  m is  set  equal  to  r 
and  (2.1)  is  solved  iteratively  for  k. 


k 


■ 
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3.  MEAN  SQUARED  ERROR  COMPARISONS 


Comparison  of  stochastic  estimators  of  the  ridge  parameter 

have  been  conducted  by  simulation  due  to  the  complexity  of  the 

theoretical  distribution  of  8 when  k in  (1.2)  is  random.  Hoerl, 

1\ 

Kennard,  and  Baldwin  (1975)  estimated  k with 


A 

k = pc/B'l  , 


where  8_  and  a are  least  squares  estimators  of  the  respective 
parameters.  They  concluded  that  8 using  the  random  estimator  of 
k offered  great  potential  for  reducing  the  mean  squared  error  of 
6 over  that  of  8_.  Following  up  on  this  study,  Hoerl  and  Kennard 

A A 

(1976)  iterated  with  (1.2)  and  (3.1)  using  k to  obtain  6.  then 

R 

A A 

inserting  B in  (3.1)  to  obtain  a new  k,  and  repeated  the  process 

K 

until  the  estimates  k (and  B^)  converged.  Their  simulation  showed 
that  the  iterated  ridge  estimator  offered  further  improvement  in 
mean  squared  error  over  least  squares  and  the  noniterated  ridge 
estimator.  In  both  of  these  studies  the  magnitude  of  the  reduction 
in  mean  squared  errors  of  the  ridge  estimators  over  least  squares 

2 

was  linked  to  the  magnitude  of  the  signal-to-noise  ratio,  p * J3'j5/a  : 
the  larger  the  value  of  p , the  less  the  reduction  in  mean  squared 
error  of  the  ridge  estimators  over  least  squares.  Also,  the  more 


multicollinear  the  data  set,  the  greater  the  reduction  in  the  mean 


squared  error  of  the  ridge  estimators. 


Dempster,  Schatzoff,  and  Wermuth  (1977)  performed  an  extensive 
simulation  involving  57  different  estimators  (including  least 
squares  and  several  ridge  estimators)  and  160  model  configurations. 
Although  no  replication  of  the  model  configurations  was  reported, 
the  ridge  estimators  generally  were  superior  to  least  squares  and 
at  least  competitive  with  the  other  biased  estimators  examined;  in 
fact,  one  ridge  estimator  (RIDGM)  was  considered  the  best  overall 
when  comparisons  were  made  with  mean  squared  error,  (1.3),  as  the 
criterion. 

Gunst  and  Mason  (1977)  compared  five  regression  estimators, 
including  least  squares  and  a ridge  estimator  using  k in  (3.1) . 

Their  simulation  involved  24  model  configurations,  each  of  which 
was  replicated  100  times.  Overall,  their  conclusions  parallel  the 
above  ones  with  two  reservations: 

(i)  there  are  specific  model  configurations  for  which 
the  ridge  estimator  is  empirically  inferior  to 
least  squares,  and 

(ii)  the  ridge  estimator  was  not  judged  the  best  overall 
although  it  was  generally  superior  to  least  squares. 

One  of  the  reasons  pointed  out  for  the  latter  conclusion  was  the 
high  degree  of  variability  associated  with  the  estimator  of  k — 
another  motivation  for  studying  nonstochastic  selection  rules  for 
k. 


f 
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We  now  wish  to  examine  the  mean  squared  errors  of  ridge  re- 
gression estimators  using  choices  of  k based  on  the  rules  RR(2) , 
RR( 3) , and  RR(4) . For  comparison  purposes,  we  calculate  mse ( Bp) 
vising  (1.3)  for  the  twelve  model  configurations  for  the  multicol- 
linear  X matrix  in  Gunst  and  Mason  (1977) . This  30  x 10  matrix 
contains  a single  strong  multicollinearity  among  the  first  four 
columns  of  predictor  variables.  The  latent  roots  of  X'X  for  the 
data  are 


l =0.00362  = 0.37145  l.  = 0.49723  H = 0.63691  =0.72105 

1 2 3 4 5 

H6  = 1.08692  f.?  = 1.22693  £g  = 1.42364  £g  = 1.55308  t1Q  = 2.47918. 


The  magnitude  of  the  first  latent  root  of  X'X  compared  with  the 
remaining  ones  indicates  that  a single  strong  multicollinearity 
exists  in  X.  The  large  elements  of  V^  identify  which  variables 


are  involved  in  the  multicollinearity: 


-1 


(0.658,  -0.487,  -0.451,  -0.353,  -0.006, 
0.001,  -0.003,  0.019,  -0.022,  -0.022). 


This  latent  vector  reveals  that  the  first  four  predictor  variables 
are  involved  in  the  multicollinearity.  The  variance  inflation  fac- 
tors (VIF)  corroborate  the  conclusion: 


Variable:  X, 


X2  X3  X4  X5 


10 


VIF:  120.1  66.4  57.1  35.9  1.2  1.2  1.3  1.6  1.2  1.6 
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The  twelve  model  configurations  incorporating  this  X matrix 

are  identified  by  all  combinations  of  four  signal-to-noise  ratios, 

2 

p = B_'B, /a  , and  three  orientations  of  B_  with-  V , 4^  = B_.  The 

four  signal-to-noise  ratios  studied  are  p = 0.04,  1.0,  100,  and 
10,000.  The  three  orientations  are  determined  by  letting  equal 

-10  (4>1  = °*0) ' °*5{Xi  + X2  + Xg  + X10}  (<^i  = °-5)  ' 311(3  Xi  (<(>1  = 

1.0).  These  choices  of  6_  enable  the  coefficient  vector  to  be 
orthogonal,  neither  orthogonal  nor  parallel,  and  parallel  to  the 
vector  defining  the  multicollinearity , respectively. 

To  eliminate  scale  differences  in  the  comparisons,  we  tabu- 

2 

late  the  scaled  mean  squared  errors,  mse(B.)/o  . For  the  ridge 
estimators  with  nonstochastic  V.  the  scaled  mean  squared  errors  can 
be  written 

P P 

mse(B_)/o2  = l l . ( l . + k)  "2  + k2p  £ U.+  k)-2^2  (3.2) 
-R  j=l  3 3 j=l  3 3 

2 2 

where  4 ^ = (V^B)  (note  that  both  V^.  and  B_  are  unit  length  vectors 
for  these  model  configurations) . Least  squares  scaled  mean  squared 
errors  are  given  by 

P 

mse(B.)/a2  = l l'1  . (3.3) 

j=l  3 


- 


. 
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4.  DISCUSSION  OF  RESULTS 

Table  1 displays  the  theoretical  mean  squared  errors  for  least 
squares  and  the  optimal  mean  squared  errors  for  ridge  regression 
for  the  model  configurations  discussed  in  the  previous  section. 

The  theoretical  value  for  least  squares  for  all  model  configura- 
tions is  given  by  (3.3);  i.e.,  for  the  latent  roots  listed  in  the 
previous  section 


10 


V = 287.63 

j=l  3 


The  optimal  mean  squared  error  for  the  ridge  estimators  is  obtained 
from  the  "generalized"  ridge  estimator 


6 = (X'X  + K)_1X'y  , 

K 


(4.1) 


where  K = diag()c^,  k^,  . ..,  k ) , of  which  the  simple  ridge  estima- 
tor is  a special  case  with  k^  * k2  * • . . = k = k.  Using  the 
generalized  ridge  estimator,  the  mean  squared  error  (1.3)  becomes 


mse 


(B_)  * I {a  i. . (£.  + k.)“2  + k'TU.  + k.)~2<t>2}  , (4.2) 

— r • DD  D j j 


2 2 

which  is  minimized  when  k^  = o /4>_. . Inserting  the  optimal  k^  into 
(4.2)  produces  the  scaled  mean  squared  errors  for  the  generalized 
ridge  estimator  that  are  exhibited  in  Table  1. 

[Insert  Table  1) 
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Simple  ridge  estimators  cannot  hope  to  achieve  the  optimal 
mean  squared  errors  shown  in  Table  1.  Nevertheless,  these  optimal 
values  provide  a gauge  of  the  maximum  reduction  in  mean  squared 
error  that  is  possible  using  ridge  regression  estimators.  Note  in 
particular  that  if  p or  4^  is  snail  enormous  reductions  are  possi- 
ble. If  both  p and  <f>j,  are  moderate  or  large,  only  small  reductions 
in  mean  squared  error  can  be  obtained;  on  the  contrary,  in  these 
situations  simple  ridge  estimators  can  perform  much  worse  than 
least  squares. 

Mean  squared  errors  for  the  ridge  estimators  RR(2),  RR(3), 
and  RR(4)  were  obtained  by  iterating  on  k until  each  of  the  criter- 
ia were  satisfied  to  four  decimal  place  accuracy  in  k.  Solutions 
for  k are 

RR(2):  k = .2100  RR(3):  k = .0172  RR(4) : k = .0165  . 

The  theoretical  mean  squared  errors  for  least  squares  and  the  ridge 
estimators  for  the  twelve  model  configurations  discussed  earlier 
are  displayed  in  Table  2 along  with  the  estimated  mean  squared 
errors  for  ridge  regression  using  (3.1)  to  estimate  k that  were 
obtained  in  the  simulation  of  Gunst  and  Mason  (1977) . The  latter 
ridge  estimator  is  labelled  RR(1)  in  Table  2 and  is  included  for 
comparison  purposes. 


[Insert  Table  2] 
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Overall  the  nonstochastic  ridge  estimators,  especially  RR(?) , 
offer  as  large  or  substantially  larger  reductions  in  mean  squared 
error  over  least  squares  as  does  the  stochastic  version,  RR(1) . 

The  notable  exceptions  to  this  statement  occur  for  p * 10,000  and 
4^  * 0.5  or  1.0.  Between  6-  and  35-fold  increases  in  mean  squared 
error  over  least  squares  are  observed  for  the  nonstochastic  ridge 
estimators  with  these  two  model  configurations.  The  stochastic 
ridge  estimator  is  comparable  in  mean  squared  error  to  least 
squares  when  p = 10,000  and  = 0.5  and  over  three  times  larger 
when  p = 10,000  and  <t>^  = 1.0. 

Conclusions  to  be  drawn  from  this  information  include  the 
following: 

(i)  except  for  extremely  large  signal -to- noise  ratios, 
nonstochastic  ridge  estimators  RR(2) , RR(3) , and 
RR(4)  offer  at  least  as  great  a reduction  in  mean 
squared  error  over  least  squares  as  does  the 
stochastic  ridge  estimator,  RR(1); 

(ii)  RR( 2)  is  overall  as  good  as  any  of  the  biased 
estimators  examined  in  Gunst  and  Mason  (1977) ; 

(iii)  when  signal-to-noise  ratios  are  large,  nonsto- 
chastic ridge  estimators  can  still  reduce  the 
mean  squared  error  over  least  squares  if  the 
coefficient  vector  is  orthogonal  to  latent  vectors 
of  X'X  that  identify  multicollinearities;  and 
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H : v;B  * 0 

O — 1 — 


vs  H : V.'B  Jt  o 
a — 1— 


with  the  statistic 


F = ^(V^  B)  /MSE  , 


(5.1) 


the  usual  normal-theory  test  statistic  for  H . Let  F denote  the 

o a 

appropriate  upper-tail  100a%  point  of  an  F distribution  with  1 and 
n-p-1  degrees  of  freedom.  Then  let 


I 

1 


■§j,T 


£ 


if 

F < F , 
a 

if 

F > F 
— a 

(5.2) 


(iv)  if  B.  is  not  orthogonal  to  latent  vectors  of  X'X 
that  identify  multicollinearities  and  if  the 
signal-to-noise  ratio  is  large,  mse (BR)  can  be 
orders  of  magnitude  larger  them  mse(B_)  . 

5.  RIDGE  PRELIMINARY  TEST  ESTIMATORS 

Since  the  only  situations  in  which  the  nonstochastic  ridge 
estimators  were  clearly  inferior  to  least  squares  in  Table  2 occur- 
red when  p was  large  and  4>^  ^ 0,  one  might  consider  the  following 
estimation  procedure.  For  X matrices  with  a single  strong  multi- 
collinearity  (the  procedure  is  readily  generalizable  to  two  or 
more  multicollinearities) , test  the  hypothesis 
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where  6^  is  a nonstochastic  ridge  estimator  (RR(2),  RR(3),  or 
RR(4)).  The  estimator  (5.2)  is  referred  to  as  a preliminary  test 
estimator  since  its  form  depends  on  the  outcome  of  the  test  of  H 

o 

Following  Bock,  Yancey,  and  Judge  (1973),  the  preliminary 
test  estimator  (5.2)  can  be  written  as 


^PT 


[o,F  ) (F) + I[F  ) (F) ' 


(5.3) 


where  1 [a  b)  (u)  * 1 if  a < u < b and  equals  zero  otherwise.  A 
straightforward  but  tedious  derivation  reveals  that 


mse<6pT)/a4 


+ kil~1(Jl1+k)"1p3(X)  (kf^+k)-1  - 2} 


P 

+ kp  (A)  l f~1U.+k)“1{k(f ,+k)”1  - 2} 
j=2  33  3 

+ pk4>^(il1+k)"1{2p3U)  - 2p5(X)  +k(fc1+kf1p5(X)} 
P 

+ Pk2p,  (X>  l U.+k)‘V  , (5.4) 

1 j=2  3 3 


where  pr(X)  ■ Pr(F(r,  n-p-1,  X)  < c/r } and  F(r,  n-p-1,  X)  is  a 
noncentral  F random  variable  with  r and  n-p-1  degrees  of  freedom 
and  noncentrality  parameter  X ■ ^ ( V' 6.)  2/2CT2 . 

The  effect  of  the  preliminary  test  on  the  mean  squared  errors 
of  the  nonstochastic  ridge  estimators  can  be  seen  in  Table  3.  To 
varying  degrees  both  preliminary  test  procedures  (using  a » 0.05 
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and  0.01)  protect  against  catastrophic  increases  in  mean  squared 
error  over  least  squares  for  p ■ 10,000  and  • 0.5  or  1.0, 
particularly  for  the  latter  orientation.  Balancing  the  decrease 
in  mean  squared  error  for  these  two  configurations  from  those  re- 
ported in  Table  2 is  a moderate  increase  in  mean  squared  error  for 
most  of  the  other  configurations.  It  is  interesting  to  observe, 
however,  that  all  the  nonstochastic  ridge  estimators  using  the  pre- 
liminary test  procedure  with  a ■ 0.01  are  competitive  with  RR(1) 
except  for  the  configuration  with  p * 10,000  and  « 0.5. 

6 . CONCLUSION 

Nonstochastic  ridge  selection  rules  are  inqportant  to  ridge 
regression  methodology  because  virtually  all  the  theoretical  prop- 
erties of  ridge  estimators  must  be  derived  under  the  assumption 
that  k is  nonrandom.  In  this  study  three  nonstochastic  selection 
rules  for  ridge  parameters  were  investigated  and  compared  with  one 
stochastic  version.  The  nonstochastic  ridge  estimators  were  shown 
to  compete  very  favorably  with  the  stochastic  one  when  mean  squared 
error  is  a criterion,  although  in  some  specific  model  configura- 
tions all  the  ridge  estimators  are  inferior  to  least  squares. 

In  the  course  of  this  investigation  several  additional  obser- 
vations were  made  that  have  a bearing  on  the  conclusions  drawn 
herein.  First,  Vinod's  (1976)  use  of  the  minimun  ISRM  (Index  of 
Stability  of  Relative  Magnitudes)  as  a criterion  for  selecting  an 
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appropriate  value  of  the  multi col linearity  allowance  performed 
erratically.  On  some  X matrices  studied  the  minimum  ISRM  indicated 
a multicollinearity  allowance  that  appeared  to  be  much  larger  than 
one  would  select  from  other  considerations  (e.g.  latent  roots  and 
vectors  of  X'X),  a tendency  that  was  also  observed  by  Wichern  and 
Churchill  (1976) . Also,  the  calculation  of  ISRM  often  indicates 
several  local  minima  and  it  is  not  clear  whether  the  absolute  min- 
imum or  the  first  local  minimum  should  be  utilized.  Our  preference 
is  to  identify  the  number  of  multicollinearities  in  X through  the 
use  of  latent  roots  and  vectors  of  X'X,  variance  inflation  factors, 
and  pairwise  correlations  among  predictor  variables  and  set  m in 
(2.1)  equal  to  the  number  of  multicollinearities  so  identified. 

A second  observation  made  during  the  course  of  this  study  is 
that  modifications  of  the  nonstochastic  selection  rules  may  be 
desirable  depending  on  the  nature  of  the  multicollinearities.  For 
example,  if  two  or  more  strong  multicollinearities  occur  in  X, 

RR(2)  can  be  less  effective  in  reducing  mean  squared  error  them 
RR (3)  or  RR(4) . This  is  because  |x'x|  can  be  so  close  to  zero  that 
a large  value  of  k is  required  to  force  |x'X  + kl | to  equal  1.0. 
Large  values  of  k do  not  reduce  the  variance  portion  of  mse(B^)  as 

K 

substantially  as  do  small  ones  and  may  thereby  negate  the  advantage 
of  using  ridge  regression. 

Finally,  modifications  of  the  preliminary  test  procedure  may 
be  possible.  These  modifications  should  be  aimed  at  reducing  the 
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1.  Least  Squares  and  Generalized  Ridge  Regression  Scaled 


Mean  Squared  Errors 


2.  Mean  Squared  Errors  of  Least  Squares  and 
Ridge  Regression  Estimators 


F 
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3.  Mean  Squared  Errors  of  Ridge  Preliminary  Test  Estimators: 
a * 0.05  (Top  Line)  and  a = 0.01  (Bottom  Line) 


Estimator 

p = 0.04 

p = 1.0 

p = 100 

p * 10,000 

a. 

<f>^  * 0.0 

RR(2) 

77.79 

77.79 

78.37 

135.72 

26.81 

26.82 

27.41 

87.19 

RR(  3) 

87.84 

87.84 

87.85 

88.30 

38.55 

38.55 

38.56 

39.02 

RR(4) 

88.28 

88.28 

88.28 

88.69 

39.09 

39.09 

39.09 

39.53 

b. 

_!i  = o.5 

RR(2) 

77.80 

78.08 

107.20 

1,330.61 

26.82 

27.11 

56.82 

2,090.71 

RR(3) 

87.85 

88.05 

108.25 

1,042.61 

38.56 

38.75 

58.20 

1,502.61 

RR(4) 

88.29 

88.48 

108.50 

1,035.02 

39.10 

39.28 

58.51 

1,488.08 

c . 

±!  = !-C 

l 

RR(2) 

77.83 

78.83 

179.26 

420.71 

26.85 

27.86 

131.40 

1,101.83 

RR  ( 3) 

87.88 

88.66 

167.77 

394.21 

38.59 

39.34 

116.67 

922.65 

RR(4) 

88.31 

89.09 

167.47 

393.29 

39'.  12 

39.87 

116.28 

916.50 
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